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ABSTRACT 



The goals of this study at Charles Sturt University 
(Australia) were to collect and interpret data about the way that students 
used various features of an electronic study guide in the context of a 
distance education setting and to uncover any relationship that affected the 
desirability of this medium of delivering educational materials. The study 
was conducted for the duration of a full semester for five consecutive 
semesters. Data were collected using computer logging procedures within the 
study materials which recorded every action along with time-based data. 
Surveys were used before the start of the subject to determine the level of 
experience with computers and computer-based learning materials. Surveys and 
interviews were used after the teaching period to gather data about student 
reactions. Data were analyzed according to the following factors: (1) a 
meaningful gain in relation to the goals of the project; (2) a payoff in 
making the life of the participant somewhat easier; and (3) a measure of the 
pleasure or satisfaction derived from the project. Findings indicated that 
there are misconceptions in the literature about how acceptable such products 
really are. Contains 17 references. (DLS) 
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Abstract: The climate of increasing demand for higher education places, serious 
discussion of global strategies for the delivery of courses and an extensive body of 
relevant literature are notable for the dearth of discussion about the effect of the 
O' use of electronic materials from a student perspective. Research which was 

5 focused on the definition of a set of features that might make electronic books a 

oo more desirable alternative to printed ones for distance education students, 

collected and interpreted data about the way that distance education students used 
Q various features of an electronic study guide. Analysis of the results indicated that 

W there are serious misconceptions in the literature about how acceptable such 

products really are. The major issues are not simply ones of how sophisticated the 
electronic book might be but ones of minimising the inconvenience to the learner. 
Many more learning features need to be provided than is generally thought 
necessary, to compensate for the restrictions that an electronic medium places on 
the learner. 



The higher education system in Australia and indeed many other parts of the world, is 
changing. The demand for more higher education places is growing as a result of community 
expectations and the effect of such things as the need for retraining [Zastrocky 1996]. 
Offering education in the distance mode is one of the key expansion areas for higher 
education. International competition and the marketing of education courses globally are 
already established trends [Ayad 1996]. For organisations that have a successful history in 
’Distance Education’ (DE) as well as those who are only just embarking on such ventures, the 
choice of most appropriate delivery medium is one of the critical factors in their success. 
Many already anticipate using electronic means such as the World Wide Web (WWW), 
CDROM etc. as a distribution mechanism. Pressures for change have been building for some 
time as successive governments cut budgets while Australian society expects progressively 
more and more of its school leavers to aspire to post secondary education. 



Teaching is a labour intensive task. No university can afford the luxury of Socratic dialogue 
for any but the most rare of students. Providing guidance, assessment and feedback for the 
vast majority is a goal that is difficult to achieve in a climate of increasing student:staff ratios 
and decreasing per capita grants. Administrators and policy makers sometimes use economic 
rationalist terms such as ’economies of scale’ or ’cost effective’ practices. There are many 
such as the vice-chancellor of the Open University in the United Kingdom, Sir John Daniel, 
[Daniel 1996] who believe that the future of higher education is inexorably tied to the 
implementation of strategies based on the use of new and emerging information technologies. 
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DE or ’Open Learning’ as it is sometimes referred to, is one of the methods by which the high 
cost of face to face teaching may be reduced. There are serious doubts about the intrinsic 
value of some of the traditional face to face methods such as the lecture, that most 
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universities have used for quite some time [Laurillard 1993], [Ramsden et al. 1995]. Prior to 
1993, the provision of distance education courses in Australia was controlled by government 
regulation but the subsequent removal of these restrictions has seen some of the larger 
universities enter the marketplace in search of DE students. The issues of cost effectiveness, 
quality of service and quality of distance education materials form a formidable set of criteria 
by which the success of DE providers will be judged. 

The appropriate use of technology is a significant component in such ventures. Computer 
technology has been used in education for many years but what is surprising, is the relatively 
superficial effect that it has had. From almost the very beginning, computers were seen as an 
important tool to be used in the educational process. Society was promised [Suppes 1968] 
the ultimate in individualised instruction, an infinitely patient tutor, a system which would 
revolutionise both the educational processes at schools and in the home. [Bork 1980] 
claimed that: "By the year 2000 the major way of learning at all levels, and in almost all 
subject areas will be through the interactive use of computers. " (p53) 

The Australian National Teaching Development Projects since 1994 bear witness to the same 
sort of confidence that Bork expressed nearly two decades ago but also highlight the 
difficulty of extremely high production costs. Such high costs are unacceptable in the current 
funding climate unless they can be spread over a large student load or produce significantly 
higher learning outcomes. The widespread adoption of the Internet and the WWW by 
academic as well as business and private interests around the world has led to suggestions 
that this is the means by which cost effectiveness may be achieved. While the concept of the 
'virtual university' is most definitely an item on the higher education agenda, questions about 
the mission of such a university [Gilbert 1996] and the impact of working in one [Taylor 
1996] are only just beginning to be raised. However, little appears to be written from the 
perspective of the learner. 

Charles Sturt University (CSU) is one of the major providers of DE in Australia with over 
24,000 students, three quarters of whom, study in the DE mode. The Technology Strategy 
Report [Rebbechi & Barnard 1994] mapped out a comprehensive set of alternatives for the 
development of technology used in the teaching and administration of CSU students. One of 
its key recommendations involved the conversion of as many as 1200 subjects from print into 
electronic form by the year 2000. This was seen as an ambitious and probably unrealistic 
target but it did indicate a support for the use of electronic teaching at the strategic planning 
level within CSU. There was general agreement that some method of automatic conversion 
would need to be used for a large percentage of these subjects and that this would not provide 
the opportunity for substantial enhancements. 

Fundamental to this decision to move to electronic format for teaching materials was the 
belief that in doing so, students would not be disadvantaged and indeed the provision of 
'added value' features would actually make this method of delivery more attractive to 
students. The adding of value is the attribute about which, least quantitative data is available. 
The notion of just what the value is, that is being added, is different from the student's, 
academic's and organisation's point of view. Quite clearly, the organisation might see value 
in the reduction of the cost of printing or the perception of 'With-it-ness' [Taylor 1996]. 
Academics might see value in the reduction of lead times from existing levels associated with 
a production model dominated by the publishing process. Students however, would not 
necessarily be aware of this type of added value or even see this as having much to do with 
the quality of their learning experiences in a course. 



For students there may be added value in just having the learning materials in electronic 
format. Such a scheme could be achieved with minimal extra effort requiring just a simple 
translation of existing DE materials that are already largely in electronic format as word 
processor files, into a suitable format for user viewing. This type of translation is already 
commonplace in the publishing industry with the use of Standardised General Markup 
Language (SGML) or commercial programs like Adobe Acrobat. 

This would facilitate access to the content in a way which allows them to manipulate, extract, 
reorganise and customise the learning materials to suit their own needs and preferences. 
Catering for individual differences and learning styles has been a long standing goal of 
education. Being able to make margin notes which do not limit one to the width of the 
margin, using copy and paste to export some sections to another electronic document or 
import sections from another source are examples of value that is added by virtue of having 
the materials in an accessible electronic form. 

Electronic copies of learning materials are usually much more compact than bulky printed 
ones, are less bulky to carry around and easily duplicated. This would allow students to have 
multiple copies of their materials at different locations such as at home and work. Taking 
study materials on a plane would add very little to the weight of the baggage and in that 
respect encourage portability. 

Whether these are sufficient to have students perceive an increase in quality is open to 
question but this represents only the minimal set of possible advantages that may accrue from 
a move to electronic format. Since cost of printing determines such things as page limits for 
learning materials, the number of colours used in the typography, diagrams and other 
graphics, these could be easily enhanced with very little increase in cost, both in terms of 
money and time. The use of colour to enhance presentation and information content is well 
documented [Macaulay 1995]. This would be an obvious example of added value from the 
student's perspective. 

The provision of more valued added items would require additional effort which would 
reduce the cost effectiveness from the organisation's perspective but if it enhances the 
learning process and results in higher success rates, better client satisfaction and ultimately an 
increase in prestige for the course, the academics or the university, then such increases may 
be justified. Among the value added features that could fall into this category are ones such 
as the better integration of resources. Providing structural features such as hotlinks, multiple 
views, expansion boxes, content maps, free text searching etc. to give the student more 
sophisticated tools with which to use the basic materials would require a significant effort on 
the part of the producers of these materials. Allowing students to make their own links is in 
the opinion of [Jonassen 1993], a vital step in constuctivist use of such learning materials. 
Links to WWW resources are an obvious extension to this type of value adding. 

[Laurillard 1985], argues that the full benefit of computer based materials does not come with 
just the student’s control over the sequence of content and learning strategy. It requires the 
provision of highly interactive simulations which are designed to help the student acquire the 
requisite knowledge by means of a goal oriented manipulation of the information in the 
domain. Such features are extremely expensive to provide and serve as the upper limit of 
what could conceivably be achieved in the process of adding value to existing materials. 

Determination of the amount of advantage or disadvantage that results from the provision of 
electronic learning materials is difficult. Even considering only the student's perspective 
there are many variations which affect the ability to make accurate judgements. Objective 



measures such as differences in the amount of time required to deal with same amount of 
content need to be analysed in the light of various factors in the students' background that 
could affect this. Allowances also need to be made for additional 'overhead time' such as 
how long it takes between the arrival of the materials and when they are actually installed and 
functional. Simple cost benefit analyses need to take account of the real cost to the student of 
such practices including hardware, software, consumables (paper, ink and possibly tele- 
communications costs) not just the obvious ones of production time and cost of distribution. 
Even so, there will be many subjective factors such as the amount of inconvenience that it 
may cause or the amount of enthusiasm that it engenders, that will be impossible to 
accurately quantify. 

Computer programming is a subject that requires a significant amount of integration of 
relatively easy to produce resources (program code as text, problems, flowchart diagrams, 
pseudocode) with the basic explanations of the concepts. Since most of these resources 
already existed in electronic form by virtue of their nature, the conversion of an introductory 
programming subject's learning materials into electronic format using a minimalist approach 
outlined above was a good choice to establish a benchmark for levels of acceptance of this 
style of delivery of DE materials. Programming was also an ideal choice because the nature 
of the subject required students to undertake significant amounts of interactive exercises with 
computers, eliminating one of the potential variables, the need to use a computer and also 
minimising the effect of some others such as computer anxiety. Programming students are 
more likely to be adopters of the technology than other students. 

A substantial body of literature is available that deals with a range of theoretical issues 
surrounding the design and implementation of hypermedia based learning materials. Most of 
it however, discusses these from the perspective of the designer rather than the end user and 
makes assumptions about student reactions. Empirical studies of users' behaviour are usually 
focused on specific features that are often investigated over a short period of time and in 
controlled laboratory situations. This study [Messing 1997] was carried out for the duration 
of a full semester for five consecutive semesters. 

The goals of the study were to collect and interpret data about the way that students used 
various features of an electronic study guide in the context of a distance education setting and 
to uncover any relationships that affected the desirability of this as a means of delivering 
educational materials. Data were collected using computer logging procedures within the 
study materials which recorded every action along with time based data. Surveys were used 
before the start of the subject to determine the level of experience with computers and 
computer based learning materials. Surveys and interviews after the teaching period were 
used to gather data about student reactions. 

The analysis of the survey data using the 3G approach proposed by [Collis 1991] proved to 
be an appropriate technique for evaluating electronic books. The '3G factors' may be loosely 
translated as: 

Gewin - a meaningful gain in relation to the goals of the project i.e. some level of 
performance or achievement 

Gemak - a payoff in making the life of the participant somewhat easier 

Genot - a measure of the pleasure or satisfaction derived from the project 

Two variations of the standard 3G approach were used. Positive and negative aspects were 
quantified separately and plotted and the whole group was divided into four categories based 
on their disposition to using the technology as well as inclination to make strong or weak 
comments. 



The 3G vector diagram agreed very well with the theoretical prediction for the use of 
electronic books in a school setting without a teacher support focus as suggested by Collis. 
One could argue that such a setting is not dissimilar to a distance education environment. 
The results in [Fig. 1] graphically illustrated the dichotomy that was evident in other aspects 
of the analysis. The positive aspects indicated a measure of achievement and satisfaction that 
students derived from using the electronic study guide while the negative components 
reflected the fact that there were some significant costs associated with using it. 




Figure 1: Modified 3G graph of student evaluations 



The gemak (payoff) vector appeared to represent the factor which needs to be addressed most 
urgently. The negative components were a measure of the obstacles that students had to 
overcome in using an electronic format which still largely restricted portability. Rather than 
making life easier, the use of the electronic study guide actually caused significant 
inconvenience. The results for the pleasure vector indicated that in general, the students in 
this study were, as suggested, adopters of the technology. As [Bates 1997] observed of other 
distance education students, they were prepared to overcome significant hurdles and still 
retain a commitment to technology-based education. 



The analysis of the usage data revealed that even though a number of significant features 
were added to the electronic book to make it a supposedly attractive alternative to the printed 
version, these were not sufficient to prevent most students from preferring the printed one. 
Electronic format places many restrictions on distance education students due largely to its 
lack of portability. It forces changes in the way that students study which are not really taken 
into account in the cost/benefit analysis that the providers of this material undertake. For 
example, one student semi-jokingly replied that "I used to read in bed at night but the 14 inch 
monitor got too heavy for my arms!”. 

Such restrictions are even more pronounced when this is coupled with tighter requirements 
such as having to connect via a telephone line. Not only does this cause disruption to family 
organisation and routine but it can cause considerable acrimony on the part of the student 
who may feel that the additional social pressure is not really being balanced by exceptionally 
good learning material. It may seem like a good move from the academic and organisational 
perspective to supply students with their learning materials over the Internet, but unless these 
materials justify the expense and inconvenience to the student, they will not be well received. 
The indications from this study are that the perceived benefits need to be much higher than 
the typical level currently available. 
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In listing the supposed benefits, one must be careful not to confuse perceived benefits from 
the designer's projection of the learning process with the reality. The way that features of 
electronic learning materials affect student learning may not be what was originally expected. 
For example, being able to easily execute program code in the examples was considered a 
highly positive aspect of providing an electronic study guide for this group of students. Data 
revealed that there were a number of unanticipated factors that partly explain why the usage 
was actually quite low. Some students who initially made moderate or extensive use 
abandoned the technique for a very sound educational reason. While the process made it 
relatively quick to demonstrate program code in action, it by-passed the act of typing code 
into the compiler. During such an act, there would normally be a great deal of incidental 
learning of the structure and syntax of Pascal programs as well as how to operate their 
compiler. These students reported feeling that they derived better learning experiences by 
actually typing the code themselves. 

The majority of students in the study could be characterised as having both highly positive 
opinions on the use of electronic format as well as highly negative ones. They could see the 
educational value of many of the features that were provided but the practical restrictions of 
having to use a computer for most of the time were dominant. They, in common with most 
DE students [Roberts 1984], use their learning materials in a diverse collection of situations, 
on the train to work, at work during lunch breaks, in planes, at airports, waiting in the car for 
a child at football training, in bed at night and many more. Not all of these situations are 
conducive for interactions with a computer. 

Electronic learning materials may well meet the various objectives that their academic 
creators have for themselves or their organisation but in the end, the test is whether the 
learners share these positive views. The implication from this study is that the break even 
point where learners prefer to use the electronic materials because of added value features 
and despite the greater restrictions, is much higher than one would expect. The following 
student comment provided an accurate summation of the general attitude: 

"It exposed us to the way all distance education is likely to he delivered in the not too 
distant future and it showed us what learning from course notes provided only in electronic 
form is really like - which wasn't as easy as I thought it would be - there is something about 
learning and reading from paper which is so much easier than doing it from a screen. I had 
originally thought that I would really enjoy studying from a computer screen - but I really 
didn't, although it was an effective way of providing a lot of interconnected information. " 



References 

[Ayad 1996] Ayad, N. (1996). Towards the virtual university: Trends and strategies. In G. 
Hart and J. Mason, Symposium proceedings & case studies of the virtual university 
symposium, Melbourne University, 91-98. 

[Bates 1997] Bates, A.W. (1997). The impact of technological change on open and distance 
learning. In Distance education, 18(1), 93-109. 

[Bork 1980] Bork, A. (1980). Interactive learning. In R.P. Taylor, (ed), The Computer in 
the School : Tool, Tutor, Tutee , Teacher's College Press: New York. 

[Collis 1991] Collis, B. A. (1991b). The evaluation of electronic books. In Educational & 
training technology international , 28(4), 355-364. 

[Daniel 1996] Daniel, J. (1996), New kids on the box: Distance education enters its third 
generation. In C. McBeath and R. Atkinson (eds), Proceedings of the 3rd International 
Interactive Multimedia Symposium, PROMACO Conventions: Perth, 1-9. 



[Gilbert 1996] Gilbert, A. (1996). The virtual university? In G. Hart and J. Mason, 

Symposium proceedings & case studies of the virtual university symposium, Melbourne 
University, 3-4. 

[Jonassen 1993] Jonassen, D.H. (1993). The future of hypermedia based learning 
environments: Problems, prospects, and entailments. In Educational multimedia and 
hypermedia annual, 1993, ed. H. Maurer, Association for the Advancement of Computing in 
Education, 270-271 

[Laurillard 1985] Laurillard, D.M. (1985). Computers and the emancipation of students : 
Giving control to the learner, in J.A. Bowden and S. Lichtenstein (eds), Student Control of 
Learning .Computers in Tertiary Education, University of Melbourne, 1-14. 

[Laurillard 1993] Laurillard, D.M. (1993). Rethinking university teaching, Routledge: London 

[Macaulay 1995] Macaulay, L. (1995). Human computer interaction for software designers. 
International Thompson Publishing: London. 

[Messing 1997] Messing, J. (1997). Learning computing concepts from electronic books, 

PhD thesis, Charles Sturt University. 

[Ramsden et al. 1995] Ramsden, P., Margetson, D., Martin, E. and Clarke, S. (1995). 
Recognising and rewarding good teaching in Australian higher education, Australian 
Government Publishing Service: Canberra. 

[Rebbechi & Barnard 1994] Rebbechi, M. and Barnard, I. (1994), Technology Strategy 
Report, Internal report, Charles Sturt University: Wagga Wagga. 

[Roberts 1984] Roberts, D. (1984). An analysis of distance education study patterns at 
Riverina College of Advanced Education. Graduate Diploma in Distance Education 
dissertation, South Australian College of Advanced Education. 

[Suppes 1968] Suppes, P. (1968). Computer technology and the future of education. In Phi 
Delta Kappan, 49,420-423. 

[Taylor 1996] Taylor, P. (1996). Virtualising universities. In G. Hart and J. Mason, 
Symposium proceedings & case studies of the virtual university symposium, Melbourne 
University, 5-12. 

[Zastrocky 1996] Zastrocky, M. (1996). The Western Governors University An ambitious 
model for "virtual" higher education. In G. Hart and J. Mason, Symposium proceedings & 
case studies of the virtual university symposium, Melbourne University, 259-262. 





U.S. Department of Education 

Office of Educational Research and Improvement ( OERI ) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 

REPRODUCTION BASTS 



This document is covered by a signed “Reproduction Release 
(Blanket) form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a “Specific Document” Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either “Specific Document” or “Blanket”). 




EFF-089 (9/97) 



